Custom API Language Models

Overview

This document details the integration process for "Custom API Language Models" with DynamoEval.

Requirements

1. The custom model endpoint must accept HTTP POST requests
2. The endpoint must handle authentication ( if required )
   • Current supported authentication types are:
     • No Authentication
     • Bearer Token
     • API Key
3. The endpoint must either:
   • Accept DynamoEval's standard request format and return the expected response format, OR
   • Use JSONata transformations to convert between your format and DynamoEval's format

Available Authentication Modes

DynamoEval supports three authentication modes when connecting to custom model endpoints.

No Authentication

No credentials are required. Use this when your endpoint does not require any authentication.

Bearer Token

A single token is sent in the Authorization header as Bearer <token>.

API Key

API Key authentication provides full control over how the credential is sent in the request headers. It has three fields:

Field	Description
Auth Header	The HTTP header name to use for authentication (e.g., Authorization, X-API-Key).
Auth Field Value	The auth scheme/token type to use (e.g., Bearer, Basic, Token). If not specified, it will be empty.
API Key	Your API key for authentication.

The resulting header is constructed as {Auth Header}: {Auth Field Value} {API Key}. If Auth Field Value is left empty, the header is simply {Auth Header}: {API Key}.

Example with Auth Field Value set:

Given:

• Auth Header: x-api-key
• Auth Field Value: Bearer
• API Key: ab-ch-ah

The resulting authentication header will be:

{"x-api-key": "Bearer ab-ch-ah"}

Example with Auth Field Value left empty:

Given:

• Auth Header: x-api-key
• Auth Field Value: (empty)
• API Key: ab-ch-ah

The resulting authentication header will be:

{"x-api-key": "ab-ch-ah"}

JSONata Transformations

Use JSONata to map between DynamoEval's standard request/response formats and your API's formats.

• Request transformation: Convert DynamoEval's request (e.g., messages, N, seq_len, temperature) into your API's expected payload.
• Response transformation: Convert your API's response into DynamoEval's expected response type (typically an array of strings).

See the examples below (e.g., "Example 2") for concrete request/response transformation expressions.

Request/Response Format Specification

Request Format

DynamoEval supports both single-turn and multi-turn conversation formats when integrating with custom language models.

Single-Turn Request Format

{
  "messages": [
    [
      {
        "role": "user",
        "content": "What is machine learning?"
      }
    ],
    [
      {
        "role": "user", 
        "content": "Explain neural networks"
      }
    ]
  ],
  "N": 1,
  "seq_len": 1024,
  "temperature": 1
}

Available fields:

• messages: Array of conversation turns, where each turn contains message objects
  • role: Must be "user" for single-turn requests
  • content: The actual message text
• N: Number of responses to generate (minimum: 1)
• seq_len: Maximum sequence length for generation
• temperature: Sampling temperature between 0-2

Multi-Turn Request Format

{
  "messages": [
    [
      {
        "role": "system",
        "content": "You are a helpful AI assistant"
      },
      {
        "role": "user",
        "content": "What is machine learning?"
      },
      {
        "role": "assistant", 
        "content": "Machine learning is..."
      },
      {
        "role": "user",
        "content": "Can you explain neural networks?"
      }
    ]
  ],
  "N": 1,
  "seq_len": 1024,
  "temperature": 1
}

The multi-turn format supports:

• Three role types: "system", "user", and "assistant"
• Conversation history within each turn
• System prompts for context setting

Response Format

Your custom model endpoint must return responses in one of these formats:

String Response Format

[
  "This is the first generated response",
  "This is the second generated response"
]

Boolean Response Format (for Guardrail Models)

[
  false,
  true
]

Code Snippets

Create Custom API Language Model

Example 1: Direct Integration (No Transformations + No Auth)

If your API already matches DynamoEval's format and doesn't require authentication:

from dynamofl.entities import AuthTypeEnum

model = dfl.create_custom_model(
    name="My Custom Model",
    remote_model_endpoint="https://api.example.com/v1/generate",
    remote_api_auth_config={
        "_type": AuthTypeEnum.NO_AUTH
    }
)

Example 2: With Transformations + Bearer Auth (Single-turn)

If your API uses a different format, you'll need to use JSONata transformations. For details about JSONata transformations in DynamoEval, refer to the JSONata Transformations section in Custom Applications documentation.

from dynamofl.entities import AuthTypeEnum

# Transform DynamoEval format to your API format
request_transformation_expression = """
{
    "messages": $reduce(messages, $append)
}
"""

# Transform your API response to DynamoEval format
response_transformation_expression = """
[choices[0].message.content]
"""
# Extract content from structured response


model = dfl.create_custom_model(
    name="My Custom Model",
    remote_model_endpoint="https://api.example.com/v1/generate",
    remote_api_auth_config={
        "_type": AuthTypeEnum.BEARER,
        "config": {
            "token": "your-bearer-token"
        }
    },
    request_transformation_expression=request_transformation_expression,
    response_transformation_expression=response_transformation_expression,
    multi_turn_support=False,
)

Connecting with known language model providers

Together AI system

If you want to connect with Together AI system, you can use the following code snippet:

from dynamofl.entities import AuthTypeEnum

endpoint = "https://api.together.xyz/v1/chat/completions"
remote_api_auth_config = {
    "_type": AuthTypeEnum.BEARER,
    "config": {
        "token": "your-bearer-token"
    }
}

# You can use your required model name here, below is an example
request_transformation_expression = """{
  "messages": $reduce(messages, $append),
  "model": "mistralai/Mistral-7B-Instruct-v0.3"
}
"""
response_transformation_expression="""[choices[0].message.content]"""

model = dfl.create_custom_model(
    name="Together AI: Custom API LM (SDK)",
    remote_model_endpoint=endpoint,
    remote_api_auth_config=remote_api_auth_config,
    request_transformation_expression=request_transformation_expression,
    response_transformation_expression=response_transformation_expression,
    
)

Databricks' LM

If you want to connect with Databricks' LM, you can use the following code snippet:

from dynamofl.entities import AuthTypeEnum

endpoint = "https://dbc-219bb6de-02df.cloud.databricks.com/serving-endpoints/databricks-dbrx-instruct/invocations"
remote_api_auth_config = {
    "_type": AuthTypeEnum.BEARER,
    "config": {
        "token": "your-bearer-token"
    }
}
request_transformation_expression = """{
  "messages": $reduce(messages, $append),
  "max_tokens": 128
}
"""
response_transformation_expression="""[choices[0].message.content]"""

model = dfl.create_custom_model(
    name="Databricks Multi-turn: Custom API LM (SDK)",
    remote_model_endpoint=endpoint,
    remote_api_auth_config=remote_api_auth_config,
    request_transformation_expression=request_transformation_expression,
    response_transformation_expression=response_transformation_expression,
    multi_turn_support=True,
)

Supported Authentication Types

The remote_api_auth_config parameter supports the following authentication types:

1. No Authentication

from dynamofl.entities import AuthTypeEnum

remote_api_auth_config = {
    "_type": AuthTypeEnum.NO_AUTH
}

2. Bearer Token

from dynamofl.entities import AuthTypeEnum

remote_api_auth_config = {
    "_type": AuthTypeEnum.BEARER,
    "config": {
        "token": "your-bearer-token"
    }
}

3. API Key

from dynamofl.entities import AuthTypeEnum

remote_api_auth_config = {
    "_type": AuthTypeEnum.API_KEY,
    "config": {
        "token": "your-api-key",
        "auth_field_name": "Basic",           # Optional, defaults to "Basic"
        "api_auth_header": "Authorization"     # Optional, defaults to "Authorization"
    }
}

All authenticated requests will include the following default headers:

{
    "Content-Type": "application/json;charset=UTF-8",
    "Accept": "application/json, text/plain, */*"
}

Authentication Header Examples

The authentication headers will be generated as follows:

• Bearer Token: Adds "Authorization": "Bearer your-bearer-token"
• API Key: Adds "Authorization": "Basic your-api-key" (customizable using auth_field_name and api_auth_header)
• No Auth: No additional headers added

Advanced Configuration

Additional parameters allow fine-tuning of the model integration:

model = dfl.create_custom_model(
    name="Advanced Custom Model",
    remote_model_endpoint="https://api.example.com/v1/generate",
    remote_api_auth_config={...},
    response_type="string",           # Type of response expected from the model (default: string)
    batch_size=32,                    # Number of requests to batch together (default: 32)
    multi_turn_support=True,          # Whether model supports conversation history (default: True)
    enable_retry=False                # Enable automatic retries on failure (default: False)
)